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Abstract 



A classical problem In mastery testing Is the choice of passing score and 
test length so that the mastery decisions are optimal. This problem has 
been addressed several times from a variety of view-points. In this paper 
the usual Indifference zone approach Is adopted with a new criterion for 
optimizing the passing score. It appears that, under the assumption of 
the binomial error model, this yields a linear relationship between optimal 
passing score and test length, which subsequently can be used In a simple 
procedure for optimizing the test length. It Is Indicated how different 
losses for both dec1S\ion errors and a known base rate can be Incorporated 
In the procedure, and how a correction for guessing can be applied. Finally, 
the results In this paper are related to results obtained In sequential 
testing and In the latent class approach to mastery testing. 

Key-words : Mastery Testing, Passing Score, Test Length, Domain-Referenced 
Testing, Decision Theory. 
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Passing Score and Length of a Mastery Test: 
An Old Proble.Ti Approached Anew 

The notion of a mastery test has arisen in the context of modem 
learning strategies such as learning for mastery and individualized 
instruction, where at several points in the instructional process decisions 
have to be made whether students have reached certain learning objectives 
or not. In most instances, this involves the administration of criterion- 
referenced tests and the use of decision rules assuming the form of passing 
scores on the test. Students with test scores exceeding the passing score 
are considered having reached the learning objectives (the "masters"); they 
are allowed to proceed with the unit or to take up a subsequent course. 
Students below the passing score (the "nonmasters") have to relearn the 
unit and to prepare for a new test. 

A usual conceptualization in the area of mastery testing is that of 
tests as samples randomly drawn Trom a domain of tasks covering a well- 
defined learning objective. Mostly, the concern is then with the propor- 
tion of correct item responses, n, say, to be expected when the entire 
domain would have been administered. Let -n^ denote the passing score on 
this domain score variable ("mastery score"), X the number of items correct, 
and c the passing score on the test. A student is a true master if n > tt 

m 

and a nonmaster otherwise, but mastery is declared if X > c and nonmastery 
if X < c. A classical problem in mastery testing is to choose a value n* 
for the test length n and a value c*.for the passing score on the test c 
such that, for a given value of -n^, the mastery decisions are optimal. 
Several authors have addressed the above problem, all using one of 
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the binomial models for relating test scores, X. to domain scores, n. Mlllman 
(1972, 1973), for example, assumes that the simple binomial model 



can be used for this purpose and provides tables which for a chosen value 
of -n^t test length, and passing score, display the probability that a per- 
son with a given domain score is classified correctly or Incorrectly. Using 
these tables, it is possible to optimize passing score and test length 
simultaneously for a selected domain score. Comparable approaches have 
been followed by Klauer (1972) and Kriewall (1972). 

FhanSr (1974) introduced the notion of an indifference zone In the 
present problem. An indifference zone arises when the mastery score, tt^^^, 
is replaced by an interval, (ttq, tt-^), so that examinees with n > tt^ are 
considered a iraster, those with n w ttq a nonmaster, and we are indifferent 
with respect to examinees with ttq < n < tt^. The interval may be taken 

symmetric about ir , but this is not necessary. For true masters and non- 
m 

masters the probability of a misclassification is largest for domain 
scores tt^ and ttq, respectively. Fhan§r proposed as a solution to choose 
the minimum value of n and a value of c for which the probabilities of 
misclassifi cation 



(1) 



P(x) = C^)Al'^) 



n-x 



(2) 



a 



n-x 



and 
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(3) 




n-x 



are not larger than preassigned values a* ind B*. This is no closed-form 



solution, and binomial tables must be entered to find the optimal values 
of n and c. It is possible to use a normal approximation, however, and in 
that case a closed-form solution is obtained (FhanSr, 1974). Wilcox (1976) 
has adopted the same eppropch and has suggested computer search routines 
using the incomplete beta function to find the solution for the binomial 
case. 

As van den Brink and Koele (1980) have pointed out, it is possible 
to correct the above solution for the possibility of guessing on multiple- 
choice or ture-false items. To perform this correction, they adopt the 
knowledge or random guessing model and simply replace the parameter rr in 
the binomial model by 



If being the domain score corrected for guessing and g the guessing param- 
9 

eter. For the latter they suggest the use of the reciprocal of the number 
of item alternatives, q*^. 

Novick and Lewis (1974) and de Gruijter (1979) present a Bayesian 
approach to the present problem extending the model with the beta dis- 
tribution as a prior for the binomial parameter u. 

As several authors (e.g., Wilcox, 1976) have, noted, different 
preassigned values a* and 3* can be selected to allow for differences in 
loss between mi sclasslfying a true master and a nonmaster. In this paper, we 



(4) 
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will take a somewhat different approach and represent possible differences 
in loss not indirectly by manipulating both probabilities of misclas- 
sification — but via the introduction of explicit parameters. But before 
doing so, we will prepare this approach and consider the case of a decision- 
maker who is indifferent to both classification errors. It appears that 
in this case there is a simple linear relation between the optimal passing 
score and test length. This can be utilized to find the solution in this 
particular case but also plays an important part in the more general case 
of different losses. 



Indifference to Both Classification Errors 



In the Wilcox solution a number P , 1/2 < P < 1, is chosan, and next 
values for n and c are determined so that the value for n is minimal and 
both.xx and 3 are not larger than P*. The fact that the same restriction 
P* is imposed on a and 3 reflects that both mi sclassifi cation errors are 
considered equally serious. But t&isxan also be expressed in another way; 
If there is equal loss in misclassifyinga. true master and a nonmaster, 
only the size of the probability of mi sclassifi cation and not the type 
of mi sclassifi cation concerns us. If so, it seems natural not to look 
for values of n and c for which (2) and (3) are both below the same 
predetermined number P* but for values for which their average 



(5) 



X=0 XstC " 



n-x 



/2 
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meets such a requirement. 

We first issume n to be fixed and look for the value of c minimizing 
the new target function defined in (5). The constant factor 1/2 may there- 
by be ignored. Adding terms to the first sum in (5) and subtracting these 
from the second yields 



(6) j (JhiXd-,)"-" 



x=c ^ 



TTJ (l-TTj) - TTq (I-TTq) 



Since the first sum is equal to 1, the value of c for which (6) is 

minimal depends only on the bracketed factor in the second sum. We knov<r 

that the binomial probability function has monotone likelihood in x (Fer- 

X n^x ' 

guson, 1967, sect. 5.2), which implies that the ratio ttj (I-ttj) / 
TrQ'^(l-TrQ)""^ is monotone increasing in x. So there is a value of. x.:for 
which-tbe sign of this factor changes from negative to positive. If we 
set c equal to this value, the second sum in (6) contains all positive 
tems and (6) is minimal. Thus, (6) is minimal for the value c obeying 



c?, »n-c* c% „ in-c* 



Logarithmizing both sides and simplifying, it appears that 
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(7) 



n 



In - 




This result is most interesting: The left-hand side is the optimal 
value of c expressed as a relative score. The right-hand side Is a constant 
which is independent of test length and only a function of the boundary 
values of the indifference zone. Thus, whenever an indifference zone is 
established, we can easilv compute (7) from its boundary values and 
immediately know the optimal passing-score for any test length . 

We now use the linear relation between c^ and n tu find an optimal 
value, n*, for the latter and thereby follow a simple procedure analogous 
to the one in the Wilcox solution. First, a number P*, 1/2 < < 1, is 
selected which serves as an upper bound to (5). The value n* is determined 
as the smallest value of n for which (5) is not larger than P*. Second, 
the ratio c*/n i? computed from the indifference zone via (7)« Third, a 
trial value for Is chosen, and (de) cumulative binomial tables are en- 
tered with this value and the iniplied value of c* to compute (5), Fourth, 
if this computation yields a value smaller than P*, a lower trial value 
for n • is selected and step three is carried out again. For values of (5) 
larger than P*, a larger trial value is selected. This proces is repeated 
until the smallest value of n is found for which (5) is not larger than 
P*. This is n*. 

In the above procedure, trial values for n may be chosen not in- 
volving an integer value for c*. As follows from (6), in that case the 
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first integer value above c* must be used. (The choice of integer value 
below c* would imply adding negative terms to the second sum in (6) making 
this suboptimal.) 

Although binomial tables for values of n up to 150 are available- 
(Ordnance Corps, 1952), tables in most text-books do not go further than 
n = 20. It is known, however, that indifference zone methods are rather 
conservative and that, for strong requirements on (5) or narrow indifference 
zones, values of n* larger than 20 can be expected. (For an impression, see 
Table 1 in Fhanfir, 1974). When longer tests are needed and no spedial , tables 
are available, one has to resort to a computer or a calculator for the above 
procedure. The programming involved is compartively simple, though, and 
some calculators posses even hardware facilities for binomial probabilities. 

Another possibility is to use an approximation to the binomial 
distribution function which is simple enough for hand calculation. A 
straightTforward approximation, based on the central limit theorem, is to 
replace (5) by 



(8) 



c/n - TTj 1 r ^0 " 
ti:Tri(l-Tri)/n]^/2 \ {i^^il-r.^/rM^'"^ 



/2, 



* denoting the standard normal distribution function. Using this normal 
approximation, we need not compute (8) completely for each trial value for 
n*. Substituting (7) into (8), it appears that this can be written as 




(9) |^$(an^/2j ^ $(bnl/2)j /2, 
with 
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-1/2 



(11) 



b = 



In 

TTptl-TTj) 



-1 



0 / 1- TTi \ 01 



The last two expressions depend only upon Jne indifference zone boundaries, 
ttq and ttj. Once a and b are calculated from these boundaries, the iterative 
procedure can be applied directly to (9). The reader who is familiar with 
the cumulative normal distribution can use well-known reference values as, 
for example, 4(-l) = .1587, $(0) = .5000, and $(1) = .8413, to s ickly 
establish whether trial values for n* meet the restriction P* imposed on 

(?). 

It is known that the normal approximation in (8) can be less accurate, 
notably when it is used for approximating tail probabilities of skew 
binomial distributions. This situation arises when both indifference zone 
boundaries are larger than .70, say, and strong requirements are -;r??osed on 
(5). A variety of better approximations are given in Molenaar (1973). When 
choosing one, we are, however, faced with a dilenma. Generally, the more 
accu**ate the approximation, the more cumbersome its calculation. Most 
approximations can be used in combination with the iterative procedure for 
test length determination only if one has access to a computer, but in that 
case the procedure can as well be carried out directly with (5). A reasonable 
accurate approximation to (5), which is not too complex, is 
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r f 1/2 1/2 1/2 l/2\ 

(12) U/(4c + 3) (1-TTj) - (4n - 4c - 1) tTj ) 

f 1/2 1/2 1/2 1/ 

*<(4n - 4c - 1) ttq - (4c + 3) (1-ttq) 



+ 



/2 



(for a discussion artd some numerical results, see Molenaar, 1973, eq. 3.20, 
pp. Ill - 114). It is recommended that this approximation be used when strong 
requirements are imposed on (5) and the choice of the indifference zone 
entails skew binomial distributions. In order to reduce the calculations, 
a good strategy Is to use (8) until it gives a solution and next to use (12) 
to find out whether it can be improved. 

As noted before, the worth of the procedure proposed in this paper 
lies in the ease with which binomial tables can be consulted. It utilizes 
a simple linear relation between optimal passing score and test length so 
that for each trial value for n* results for only one passing score need to 
be obtained. In the other indifference zone methods several trial values for 
c must be tested for each trial value for n until t ie combination (c , n ) 
meeting the requirements Is found. The fact that c* has a simple relation to 
n gives (7) value in its own right. It can be used, for example, to find 
^^/ptimal passing scores on new tests when test length has to be fixed for 
some practical reason, or to establish whether passing scores that have 
already been used in practice satisfy the optimality condition considered 
in this paper. As will be shown In the next section, another advantage of 
the present procedure is the possibility of incorporating different losses 
for false positive and false negative decisioriie 
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Different Losses for Both Decision Errors 



So. far it has been assumed that the loss incurred for a false positive 
decision (granting the mastery status to a nonmaster) is equal to the loss 
for a false negative decision (granting the nonmastery status to a master). 
We now assume that both losses take m different values and incorporate this 
in the procedure by replacing the average in (5) by the weighted average 



(13) 



c-1 



^ x=0 ^ " x=c ^ " " 



n-x 



where Iq 'is the loss of raisclassifying a nonmaster and of a master. 

Follov^ing the same derivation as before, (13) is minimal for the value 
c* given by 



l-ir. 



In 



In X 



(14) 



l-TT, 



n 



X denoting the loss ratio Aq/^-i- 

Comparing (14) with (7), several things can be noted: The right-hand 
side of (14) displays an additive structure consisting of two different 
parts. The first part is equal to (7), and thus again a constant dependent 
only upon the indifference zone boundaries; the second part represents the 
influence of the loss r?itio on the optimal passing score and is, as opposed 
to th^ first part, dependent on tast length. When different losses for 
both raisclassifications have to be reckoned with, the optimal passing 

t5 
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score (expressed as a relative score) is thus equal to the one for the 
case of equal losses plus a test length dependent correction. For loss 
ratios larger than one this correction is positive, while it is negative 
for ratios smaller than one. 

It should be noted tnat in the second term of (14) test length figures 
only in the denominator. This implies that the longer the test is the 
smaller the absolute size of the influence of the loss ratio on the optimal 
relative passing score will be. Table 1 shows this for loss ratio values 



Insert Table 1 about here 



from 1:4 to 4:1. For example, the relative passing score on a 10- Item test 
must be raised by .173-to account for a loss ratio X = 3, while this is 
only .035 for a 50-itMi test. 

In view of the determination of optimal test length, it is helpful 
to rewrite (14) into 



In 



(15) 



(l-^l) 



In X 



n + — 



iro(l-7ri) TrQ(l-Trj) 



This expression again shows a linear relation between c" and n. It has (7) 
as slope and this time a non-zero intercept which is a function of the loss 
ratio. Table 2 shows values of this intercept for loss ratio values from 



Insert Table 2 about here 



Passing Score and Length 
13. 



ERIC 



1:4 to 4:1 and indifference zones which can often be encountered in the 
practice of mastery testing. The entries in this table are thus to be 
added to the optimal passing score for the equal loss case, c*, when loss 
ratios unequal to one are used. 

The determination of optimal test length proceeds along the same lines 
as in the previous section. First, the number P" is selected as the upper 
bound to (13). Its minimum value is no longer equal to 1/2. (In the 
previous section, this value could always be realized by randomly assigning 
the examinees to the mastery and nonmastery state.) Now it is equal to 



'46) max 



|v(Ao+h)» V^V*i)j» 



these two values being obtained by always assigning the examinees to the 
mastery and nonmtstery state, respectively. Second, the slope and inter- 
cept in (15) are computed. (For the latter Table 2 can be used.) Third, a 
trial value for n", the optimal test length, is chosen, and the associated 
value of c" is computed from (15). Binomial tables are entered with the 
value of c" to obtain the (de)cumulative probabilities in (13), and once 
these are found (13) is computed. Fourth, the va^ue computed for (13) is 
compared with P". If it is smaller (larger) than P", a lower (larger) trial 
value for n° is selected, and the previous step is repeated. The process 
is stopped when the sraalTest value of n is met for which (13) is not larger 
than P°. This is n''. 

When a normal approximation is needed, we replace (8), analogous to 
(13), by a weighted average of (de)cumulative normal probabilities. 
Substitution of (14) in this new target function results in 



.17 
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(17) 



(^0 + V 



-1 



with a and b given by (10) and (11), respectively, and c and d by 



(18) 







-1/2 


[in ^I^^-^O)] 


-1 


c = 


Trj(l-Trj_) 




In X 






TrQ(l-Trj)^ 





and 



(19) 



d = 



-1/2 



TrQ(l-Trj) 



In X 



In the above procedure, we first compute a, b, c, and d from ttq, ttj, 
and A, and next substitute our trial values for n** directly into (17). If 
necessary, we can use the approximation in (12) to find out whether the 
solution thus obtained can be made more accurate. 



Incorporating a Known Base Rate 



If a priori knowledge about the proportion of masters is available, 
for instance, from previous testing programs or experiences with comparable 
groups of students, it may be prudent to incorporate this in the decision 
procedure as well. 

Ignoring the examinees in the indifference zone for a while, let ii 
dertote the proportion of masters so that 1 - y equals the proportion of 
nonraasters. We now use y and 1 - y as weights in our target function and 
replace (13) by 
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(20) 



^ x=0 ^ ^ 



fix X/, xn-x 



x=c 



pAj + (l-y)AQ 



-1 



Follm/.ing the same derivation as before, the optimal passing score c' 
proves to be given by 



(21) 



I-iTq 1-p 
In — ^ In X In — 

l-TTj y 



IniiilV „^„!l(^ 

T^q{^-"i) T^q{^-"i) -"Qi^-T^l) 



This result is equal to (14) extended with a term containing the base 
rate, (1 - p)/p. For base rate values larger than one, this term is 
positive, while Is Is negative for values smaller than one. 

The roles played by (l-p)/p and X in (21) are fully identical. For 
a quantitative Impression of the last term in (21), Tables 1 -2 can be 
consulted with (l-p)/p substituted for X. 

To find the optimal test length in the p'^esent case with an explicit 
base rate, the same procedure as In the previous section can be followed. 
Even the same formulae (and Table 2) may be used. This stems from the 
fact that the last two terms in (21) can be reducad to the same denominator, 
whereupon (21) has the same structure as (14). The only modifications 
needed are the substitution of X(l-p)/p for x and the replacement of (16) 
by 



(22) max^ (l-y)AQ/I(l-p)AQ+vJij] , pAiiE(l-p)Ao+pAi] 

t9 



Passing Score and Length 
16. 



which now is the minimum value of the upper bound P* to (20). 

Guessing 

As noted earlier, van den Brink and Koele (1980) proposed to use the 
knowledge or random guessing model to correct Fhan§r's (1974) approach for 
the possibility of guessing on multiple-choice or true-false items. The 
same can be done in the approach given in this paper. We then first 
establish the indifference zone on the ab lity scale corrected for guessing, 
i.e., as (TTgQ, TTgj), and next apply transformation (4) to obtain the 
values (ttq, ttj) with which we enter the formulae given in this paper. 

It should be noticed, however, that experience with the knowledge or 
random guessing model in item response theory shows guessing parameter 
values somewhat less than the reciprocal of the number of alterr^tives, 
q ^ (Lord; in press). For example, items with four alternatives typically 
result in values of .22 or .23 rather than .25. It is recommended that 
this be taken into account when setting the guessing parameter value. 

Discussion 

The results presented in this paper relate to results obtained in two 
other areas. 

The first is the tent class approach to mastery test4ng. In this 
approach it is assumed that mastery and nonmastery are two latent states 
underlying the test score, each entailing different probabilities of a 
successful reply to the items. Im Emrlck's latent class model (Davis, 
Hickman, and Novick, 1973, pp. 32-47; Emrick, 1971; Emrick and Adams, 1969; 
Fricke, 1974; Macready and Dayton, 1977) two latent success probabilities 
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are assumed, one representing the mastery and the other the nonmastery 
state. Emrick and Adams (1969) give an optimal passing score which, al- 
though derived and presented in a different way, is quickly seen to be 
equivalent to c' given in (21). 

This equivalence is only formal, however. In Emrick 's model the latent 
success probabilities, w&ich correspond with iTq and in (21), must be 
estimated from the test data. (For a review of available estimation proce- 
dures, see van der Linden, 1980,). In this paper, and represent no 
latent classes and need not be estimated; they are boundary values of an 
indifference zone on the domain^score continuum which are set on educational 
grounds . 

Fricke (1974) has given proofs that the correction of Emrick 's passing 
score needed for loss ratio and base rates unequal to one are independent 
of the base rate and the loss ratio, respectively, and of the test length. 
However, this follows immediately from inspecting the structure of (21) 
which can be viewed as a linear decomposition cf c'. Van der Linden (1978) 
has proposed a correction for guessing for Macready and Dayton's (1977) 
version of Emrick 's model which corresponds with the correction for guessing 
proposed in the previous section. 

The formal correspondence between Emrick 's passing score and (21) 
suggests the use for Emrick 's model of the procedure for test length 
optimiiation developed in tbis paper. The only difference is then, of course, 
that the success parameters ttq and Uj, in (20) - (21) must be estimated 
before the procedure can be applied and that, consequently, no exact but 
estimated results are obtained. 

The second area to which the results in this paper relate is Wald!s 
sequential probability ratio test for binomial populations. Several expres- 
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sions in Wald (1947) are reminiscent of the formulae given in this paper. 
For example, formula (15) is equivalent to the critical numbers in the test 
of n < ttq against tt 5 ttj (Wald, 1974, eqs. 5.1 — 5.2). The only exception 
is that the loss ratio X is replaced by a ratio based on the probabilities 
of errors of type I and II. It roust be borne in mind, however, that, just as 
in the previous case, this equivalance is only formal and that different 
interpretations are involved. In sequential .testing test length, or, 
generally, the number of observations, is a random variable, and sampling 
is not stopped until one of the critical numbers is exceeded. The purpose 
of this paper was to find an optimal test length which is fixed prior to the 
test administration. It should be; realized, however, that when sequential 
testing strategies are possible this is certainly worth considering, since 
substantial savings in the number of test items needed can be expected 
(Wald, 1947, sect. 3.6). 

As a final coranent, we recall that the use of the binomial model 
imposes certain restrictions, on the test items. These are, however dependent 
on whether the item responses can be viewed as deterministic or stochastic 
responses (van der Linden, 1979). Either conception of item responses 
involves equal item difficulties. But, when the former conception is adopted, 
this condition can be avoided by giving separate samples of items to each 
student, whereas this cannot be done for the latter. When these conditions 
are not met, the simple binomial model us3d In this paper must be replaced 
by the compound binomial model. It is known, however, that this entails a 
smaller variance of test scores, so that the procedure of this paper will 
result in conservative estimates of optimal test length. 
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TABLE 1 

Increase of Optimal Relative Passing Score Produced by 
Loss Ratios Unequal to 0ns for Some Values of 
n and (tt^, tt,) = (.75, .85) 



n 


.25 


.33 


.50 


1 


2 


3 


4 


10 


-.218 


-.173 


-.109 


.000 


.109 


.173 


.218 


20 


-.109 


-.086 


-.054 


.000 


. .054 


.086 


.109 


30 


-.073 


-.056 


-.036 


.000 


.036 


.056 


.073 


40 


-.054 


-.043 


-.027 


.000 


.027 


.043 


.054 


50 


-.044 


-.035 


-.022 


.000 


.022 


.035 


.044 


60 


-.036 


-.029 


-.Qi8 


.000 


.018 


.029 


.036 
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TABLf^ 2 

Increase of Optimal Passing Score Produced by Loss Ratios 
Unaqual to One for Some Indifference Zones 







.25 


.33 


.50 


1 


2 


3 


4 


.60. 


.65) 


-6.491 


-5.144 


-3.245 


.000 


3.245 


5.144 


6.491 


.60, 


.70) 


-3.138 


-2.486 


-1.569 


.000 


1.569 


■2.486 


3.138 


.60, 


.75) 


-2.000 


-1.585 


-1.000 


.000 


1.000 


1.585 


2.000 


.65, 


.70) 


-6.073 


-4.813 


-3.037 


.000 


3.037 


4.813 


6.073 


.65, 


.75) 


-2.891 


-2.291 


-1.445 


.000 


1.445 


2.291 


2.891 


.65, 


.80) 


-1.807 


-1.432 


-0.903 


.000 


0.903 


1.432 


1.807 


:.7o. 


.75) 


-5.516 


-4.371 


-2.758 


.000 


2.758 


4.371 


5.516 


[.70, 


.80) 


-2.572 


-2.038 


-1.286 


.000 


1.286 


2.038 


2.572 


:.7o. 


.85) 


-1*562 


-1.238 


-0,781 


.000 


0.781 


1.238 


1.562 


[.75, 


.80) 


-4.819 


-3.819 


-2.409 


.000 


2.409 


3.819 


4.819 


(.75, 


.85) 


-2.180 


-1.727 


-1.090 


.000 


1.090 


1.727 


2.180 


(.75, 


.90) 


-1.262 


-1.000 


-0.631 


.000 


0.631 


1.000 


1.262 


(.80, 


.85) 


-3.980 


-3.154 


-1.990 


.000 


1.990 


3.154 


3.980 


(.80, 


.90) 


-1.710 


-1.355 


-0.855 


.000 


0.855 


1.355 


1.710 


(.80, 


.95) 


-0.890 


-0.705 


-0.442 


.000 


0.442 


0.705 


0.890 
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